Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ontology Cache: Ignore third-party properties consistently #183

Merged
merged 12 commits into from
May 14, 2020

Conversation

tobiasschweizer
Copy link
Contributor

@tobiasschweizer tobiasschweizer commented May 13, 2020

@tobiasschweizer tobiasschweizer added the bug Something isn't working label May 13, 2020
@tobiasschweizer tobiasschweizer self-assigned this May 13, 2020
@tobiasschweizer
Copy link
Contributor Author

@subotic I need your help to set up the supported release of Knora on github-ci.

@tobiasschweizer
Copy link
Contributor Author

@subotic I also have problems getting the zip file, giving up for today ...

@subotic
Copy link
Contributor

subotic commented May 13, 2020

Maybe this is something @benjamingeer could take a look at? knora-api should be running at that time. I don't know why the ziped file is not sent correctly. Shouldn't the curl be called with a flag to fail if something other than 200 is returned?

@subotic
Copy link
Contributor

subotic commented May 13, 2020

In one of my branches the curl is called with --fail and the error returned was 500 Internal Server Error: https://github.com/dasch-swiss/knora-api/pull/1644/checks?check_run_id=668345491#step:9:921

@subotic subotic marked this pull request as draft May 13, 2020 18:36
@benjamingeer
Copy link
Contributor

@subotic

In one of my branches the curl is called with --fail and the error returned was 500 Internal Server Error

I would need to see the Knora stack trace.

@benjamingeer
Copy link
Contributor

Maybe this is something @benjamingeer could take a look at? knora-api should be running at that time. I don't know why the ziped file is not sent correctly.

The only way to find out would be to print out the response from Knora as well as any stack trace on the Knora console.

@subotic
Copy link
Contributor

subotic commented May 13, 2020

@benjamingeer Hmm, yes, the stack trace would be useful. You are good with bash, could you maybe write a wrapper bash script for curl that calls make stack-logs-api-no-follow in the case of a non 200 status?

@tobiasschweizer
Copy link
Contributor Author

@subotic Aside from the zip problem, could you help me set up a tagged Knora release?

@tobiasschweizer
Copy link
Contributor Author

I tried to request the zip file locally and got

{"error":"akka.stream.BufferOverflowException: Exceeded configured max-open-requests value of [64]. This means that the request queue of this pool (HostConnectionPoolSetup(0.0.0.0,3333,ConnectionPoolSetup(ConnectionPoolSettings(15,0,3,64,1,Duration.Inf,100 milliseconds,2 minutes,30 seconds,ClientConnectionSettings(Some(User-Agent: akka-http/10.1.11),9999 milliseconds,479999 milliseconds,512,None,WebSocketSettings(,ping,Duration.Inf,akka.http.impl.settings.WebSocketSettingsImpl$$$Lambda$384/0x0000000800481040@66d39d4b),List(),ParserSettings(2048,16,1024,64,8192,64,8388608,8388608,256,2097152,Strict,RFC6265,true,Set(),Full,Error,Map(If-Range -> 0, If-Modified-Since -> 0, If-Unmodified-Since -> 0, default -> 12, Content-MD5 -> 0, Date -> 0, If-Match -> 0, If-None-Match -> 0, User-Agent -> 32),false,true,akka.util.ConstantFun$$$Lambda$260/0x00000008003c9040@5be5d970,akka.util.ConstantFun$$$Lambda$260/0x00000008003c9040@5be5d970,akka.util.ConstantFun$$$Lambda$261/0x00000008003ca040@9a5c703),None,TCPTransport),New,1 second),akka.http.scaladsl.HttpConnectionContext$@7310b19c,akka.event.MarkerLoggingAdapter@6c60acf6))) has completely filled up because the pool currently does not process requests fast enough to handle the incoming request load. Please retry the request later. See http://doc.akka.io/docs/akka-http/current/scala/http/client-side/pool-overflow.html for more information."}

I worked the second time I tried.

@tobiasschweizer
Copy link
Contributor Author

@benjamingeer @subotic I gave it a lot of Valium (432e98c) and now the test passed.

@tobiasschweizer
Copy link
Contributor Author

@flavens now third party props like rdfs:label are consistently excluded from the entity info on a ReadResource. This should make the error message in the GUI (UI lib) disappear. I built the lib locally with yalc and integrated it into the UI lib. It works for me. It would be great if you could confirm that it works correctly.

  1. check out wip/DSP-175-third-party-props into your local git repo of JS lib
  2. run npm run yalc-publish from the project root
  3. from your UI lib repo run yalc add @knora/api and npm i
    (like the way we did before we had the RC on npm)

In a next step, we can do https://dasch.myjetbrains.com/youtrack/issue/DSP-142 to move the logic from the UI to the JS lib.

@flavens
Copy link
Contributor

flavens commented May 14, 2020

[...] It works for me. It would be great if you could confirm that it works correctly.

@tobiasschweizer I confirm it works, no more console.error displayed and no more third-party prop in the given entityInfo object (only ResourceProp and SystemProp)

@tobiasschweizer tobiasschweizer marked this pull request as ready for review May 14, 2020 12:25
@tobiasschweizer tobiasschweizer requested a review from flavens May 14, 2020 12:25
@benjamingeer
Copy link
Contributor

@subotic

You are good with bash

I'm just good at copying and pasting things from StackOverflow. :)

could you maybe write a wrapper bash script for curl that calls make stack-logs-api-no-follow in the case of a non 200 status?

Yes, but for now could we just add make stack-logs-api-no-follow to the GitHub action here?

@tobiasschweizer

Exceeded configured max-open-requests

Maybe the problem is that all the test data requests are being done concurrently, and there are too many of them for Knora's incoming connection pool. @subotic do you know the best way to deal with this in Akka?

@tobiasschweizer
Copy link
Contributor Author

@subotic

You are good with bash

I'm just good at copying and pasting things from StackOverflow. :)

could you maybe write a wrapper bash script for curl that calls make stack-logs-api-no-follow in the case of a non 200 status?

Yes, but for now could we just add make stack-logs-api-no-follow to the GitHub action here?

@tobiasschweizer

Exceeded configured max-open-requests

Maybe the problem is that all the test data requests are being done concurrently, and there are too many of them for Knora's incoming connection pool. @subotic do you know the best way to deal with this in Akka?

It would be great if you could find out about this because it makes the test framework unstable. Maybe it's also a problem with the resources available on github-ci. I think you also experience problems for the knora-api tests, right?

@tobiasschweizer
Copy link
Contributor Author

I might get away with my curl options:

0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0
136
0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0
137
0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0
138
0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0
139
Warning: Transient problem: HTTP error Will retry in 1 seconds. 5 retries
140
Warning: left.
141

142
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
143
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
144
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
145
0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
146
0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0
147
0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0
148
0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0
149
0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0
150
0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0
151
0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0
152
0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0
153
0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0
154
17 83704 17 14247 0 0 1179 0 0:01:10 0:00:12 0:00:58 2923
155
100 83704 100 83704 0 0 6930 0 0:00:12 0:00:12 --:--:-- 21617
156
http_codesleep 120

@benjamingeer
Copy link
Contributor

Maybe it's also a problem with the resources available on github-ci. I think you also experience problems for the knora-api tests, right?

The problem we have in the knora-api tests is that GraphDB runs out of memory, that's something different.

@subotic I think that in this case, we could just do the requests one at a time, there's no reason to do them concurrently. What do you think is the easiest way to queue them?

@tobiasschweizer
Copy link
Contributor Author

https://youtu.be/yfl6Lu3xQW0?t=8

got away with it ...

@tobiasschweizer
Copy link
Contributor Author

@flavens Is this PR good to go?

Copy link
Contributor

@flavens flavens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tobiasschweizer tobiasschweizer merged commit 81169a6 into master May 14, 2020
@tobiasschweizer tobiasschweizer deleted the wip/DSP-175-third-party-props branch May 14, 2020 14:08
@subotic
Copy link
Contributor

subotic commented May 14, 2020

@benjamingeer I will create a separate test in knora-api for this. We can then move the discussion to there.

@tobiasschweizer
Copy link
Contributor Author

@benjamingeer I will create a separate test in knora-api for this. We can then move the discussion to there.

sounds good, thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants